-
Notifications
You must be signed in to change notification settings - Fork 540
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Init: pypa/installer #700
Merged
Merged
Init: pypa/installer #700
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
groodt
force-pushed
the
groodt-init-installer
branch
from
May 14, 2022 03:36
d7cb603
to
b4acf79
Compare
thundergolfer
requested review from
thundergolfer
and removed request for
brandjon and
lberki
May 15, 2022 07:29
thundergolfer
approved these changes
May 20, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Appreciate the "other information" section.
What’s the process for merging @thundergolfer ? |
@groodt I can merge it. I guess @thundergolfer was waiting for the review comments to be addressed. I checked they are fine. Thanks for addressing them. |
12 tasks
ianoc
pushed a commit
to ianoc/rules_python
that referenced
this pull request
Jun 1, 2022
This reverts commit 9b8ab1e.
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
Please check if your PR fulfills the following requirements:
PR Type
What kind of change does this PR introduce?
What is the current behavior?
The
Wheel
related parsing / unpacking / installation is using custom written code and the pkginfo package to parse metadata. This logic is complex and fragile because of the many Python Packaging PEPs it relates to.Issue Number: #671
What is the new behavior?
This PR introduces a modern, PEP standards-compliant library called installer. This package is maintained in the Python Packaging Authority and is written by a few of the current maintainers of
pip
. It is therefore more likely to be processing Wheel files correctly.This PR is only using installer for metadata parsing, but if this PR is accepted, I hope to replace all of the custom Wheel processing code.
Does this PR introduce a breaking change?
Other information
There are modern Python packaging libraries and PRs emerging that these Bazel rules should adopt where appropriate.
Modern packaging libraries:
installer from this PR, is able to correctly "install" (or rather unpack) Wheels into the desired location.
build is able to replace the elements of
pip
that produces Wheels. Suitable for converting ansdist
into a Wheel.hatch is a modern, extensible Python project manager that is a good example of how best to use the modern and emerging Python Packaging standards.
Promising PRs:
pip metadata-only resolve with pip download --dry-run --report when this PR lands and becomes standardised, the new internal resolver of
pip
can be used to produce an approximation of a lockfile. Note: this is not a formal specification of a lockfile (which will require a PEP), but is a promising step in that direction. This would eventually remove the need forpip-tools
.All of these things together means that there will eventually be the following options for consuming third-party dependencies with
rules_python
.Option A - Install pre-built Wheels only in WORKSPACE (similar to rules_jvm_external where Bazel only needs to download and unpack)
pip wheel
pip download --dry-run --report
and create.bzl
file.bzl
and unpack using installerOption B - Build and install in WORKSPACE (approach in this PR where Bazel does a combined build and unpack of third-party code)
pip wheel
and the fragile "build on demand" nature ofpip
to resolve and download existing Wheels OR convertsdist
to WheelsOption C - Something like rules_pycross (which attempts to fuse python package builds into Bazel as closely as possible)
See: https://github.com/jvolkman/rules_pycross
Option D - Something like dbx_py_pypi_piplib (where you build cpython and libraries form pypi in Bazel)
See: https://github.com/dropbox/dbx_build_tools